• Importing lm_eval currently loads all 27+ model backends at startup, even when you only need one. This pulls in all heavy dependencies (transformers, torch, vllm, anthropic, etc.).

• pip install lm_eval no longer installs the HF/torch stack.

Changes

Rewrote the registry to store lazy placeholders (lm_eval.models.huggingface:HFLM) instead of eagerly importing model classes. Models are only materialized when you actually call lm_eval.api.registry.get_model("<alias>").

Key bits:

• Registry[T] generic class with thread-safe lazy materialization
• Double-check locking pattern to avoid race conditions
• LRU cache on the materializer to skip repeated imports
• Backward compat: old MODEL_REGISTRY dict still works, decorators still work

lm_eval/models/__init__.py now just registers string paths:

  MODEL_MAPPING = {
      "hf": "lm_eval.models.huggingface:HFLM",
      "vllm": "lm_eval.models.vllm_causallms:VLLM",
      ...
  }

Breaking changes

• Code that accessed submodules via attribute (lm_eval.models.huggingface.HFLM) breaks - you need an explicit imports:

from lm_eval.models.huggingface import HFLM # works

 import lm_eval.models
 lm_eval.models.huggingface.HFLM #  AttributeError

• Moved torch, transformers, accelerate out of core deps. Install what you need:

  pip install lm_eval          # core only, no model backends
  pip install lm_eval[hf]      # huggingface backend
  pip install lm_eval[vllm]    # vllm backend
  pip install lm_eval[api]     # openai/anthropic/etc